Skip to content

spike: Phase 4 — measure parallel rendering, decide against sharding - #21

Open
TheMeinerLP wants to merge 2 commits into
clean2/phase-3-hostingfrom
clean2/phase-4-sharding
Open

spike: Phase 4 — measure parallel rendering, decide against sharding#21
TheMeinerLP wants to merge 2 commits into
clean2/phase-3-hostingfrom
clean2/phase-4-sharding

Conversation

@TheMeinerLP

Copy link
Copy Markdown
Contributor

Note: This replaces #13, stacked on clean2/phase-3-hosting instead of clean/phase-3-hosting, and squashed on a base where the spike entrypoint's environment-variable validation already uses the require_env helper, so the squash commit itself never contains the shell idiom the scanner misread as a secret.

Replaces #5, closed. Same content, rebuilt as a single squash commit on clean/phase-4-sharding because a secret scanner flagged disposable test credentials in the old commit history and history cannot be rewritten in this environment. See the closing comment on #5 for detail.

Phase 4 of Apus was gated on a spike. It ran, it came out negative, and this PR records that plus the resulting decision. No sharding is implemented, deliberately.

The question

Can two BlueMap processes render disjoint region sets of the same map into the same map storage without damaging the zoomed-out views?

Render state is stored granularly per tile and chunk, so disjoint regions do not collide there. But lowres levels are built by aggregation — a lowres tile averages colour, height and light across a group of higher-resolution tiles, and those groups span region boundaries. Earlier research called conflicts "unlikely" because lowres tiles are also stored granularly. That reasoning conflates two different things, which is why this was measured rather than assumed.

The measurement

A single-pass reference render of the whole world, against two concurrently running containers rendering disjoint, adjacent region sets into a second bucket.

  • 7 of 24 lowres tiles diverge from the reference, reproduced identically three times.
  • One tile drops from 99 % rendered terrain to 91 % empty.
  • A sequential control run — no race at all — corrupts 10 of 24 tiles. Order dependence confirms the mechanism independently of timing.

Granular storage prevents corruption of individual tiles. It does not prevent two shards overwriting the same aggregated lowres tile.

Full report with setup, raw numbers and an honest assessment of what the experiment does and does not cover: docs/superpowers/spikes/2026-08-09-lowres-sharding-spike.md.

The decision

The spec offered two fallbacks. This picks the second — vertical scaling via render-threads — over the two-stage alternative (shards render hires only, a final pass builds lowres):

  • The two-stage route needs a custom runner bound to BlueMap-Core. Section 1.4 rules that out for the MVP, and 2.1 explains why: BlueMap-Core is not a stable public API.
  • Vertical scaling already exists and costs nothing.
  • No world has yet taken long enough to justify the work.

The architecture stays sharding-capable — the region list is in the bundle manifest, BlueMapMap.spec.shards exists. If a world ever does take too long, the two-stage approach is what to evaluate, and this spike is the ground to build on. Until then shards stays at 1.

Limits of the experiment, as stated in the report

Small test world, render-mask used to split rather than the scheduleMapUpdateTask API a real implementation would use, and only one split topology tested.

Phase 4 was gated on a spike into whether two BlueMap processes can render
disjoint region sets of the same map into the same map storage without
damaging the zoomed-out (lowres) views. No sharding is implemented,
deliberately.

- Lowres levels are built by aggregation across region boundaries, unlike
  granularly stored render state. Measured directly: 7 of 24 lowres tiles
  diverge from a single-pass reference render when two containers render
  disjoint adjacent region sets concurrently, reproduced identically three
  times; a sequential control run with no race at all still corrupts 10 of
  24 tiles, confirming order dependence rather than a timing artifact.
- Full report with setup, raw numbers and an honest assessment of scope:
  docs/superpowers/spikes/2026-08-09-lowres-sharding-spike.md. The spike's
  own entrypoint script validates its environment variables with the same
  single-mention helper function as the Phase 1 runner image.
- Decision: scale vertically via render-threads instead of the two-stage
  alternative (shards render hires only, a final pass builds lowres) —
  the two-stage route needs a custom runner bound to BlueMap-Core, which
  is not a stable public API, and no world has yet taken long enough to
  justify the work. The architecture stays sharding-capable (region list
  in the bundle manifest, BlueMapMap.spec.shards exists) but shards stays
  at 1 until a world actually needs it.

This branch replaces clean/phase-4-sharding (PR #13), stacked on
clean2/phase-3-hosting instead of clean/phase-3-hosting, and squashed on a
base where the spike entrypoint's environment-variable validation already
uses the helper function, so no follow-up fix commit is needed to clear the
scanner finding.
@TheMeinerLP
TheMeinerLP force-pushed the clean2/phase-3-hosting branch from cb571e5 to 0a64b11 Compare August 9, 2026 10:30
@TheMeinerLP
TheMeinerLP force-pushed the clean2/phase-4-sharding branch from 570d358 to ddd0c69 Compare August 9, 2026 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant